If it dropped after the ball exited the court, it is your point.
If it happened before, you lose the point, because you are not allowed to enter opponent's court with your body, racket, or any other object. Regardless of you touching the net or not.
Also, independent of your situation, you cannot just keep dropping something on the court, saying "oopsie", and playing let afterwards. First time it drops by accident, it is involuntary inference and it is a let (unless it drops to the opponent's side, in your example). If it happens for a second time, it is assumed voluntary, because you did not prevent it from happening (by tightening your cap or leaving it out of the court), and the point is automatically awarded to the opponent.
This is valid for competitive matches. People will be a bit more understanding in an amateur setting.