A wormhole picket is a cloaked ship that is used to monitor a wormhole for alien traffic.
Game Mechanics Used
When a wormhole is traversed, any ship on grid with the wormhole and looking in its direction will see it flash (which is somewhat subtle). Also, regardless of camera angle a distinctive noise will be heard. I refer to this noise as the "wormhole noise", and often try to transcribe it as "bwoomp-crackle". (Here is a youtube video about it, although unfortunately recorded with music on.) Wormhole noise will vary in loudness depending on how far your camera's point of view is from the wormhole. You can hear it pretty far off if your environment is quiet, but I prefer to be very close so I can play music and not miss it.
Since you don't need to be watching to hear noises, you can monitor the wormhole without actually looking at the screen it is being displayed on. The client can be minimized or underneath, although this makes it harder to see who came in before they cloak. Indeed, you can monitor a wormhole from across the room.
Setting up a Picket
For a picket to work, be sure your sound is working. Assuming that is the case, setting up a picket is quite simple. Your goals here are as follows:
(1) be on grid with the wormhole, within "Look at" distance (100km)
(2) be as far from the wormhole as possible, subject to (1), convenience, and any offensive desires
(3) be cloaked
Of those, (1) and (3) are pretty self-explanatory. Why (2)? Because there is always a small chance of being uncloaked by people zooming off the wormhole. And there is also always some chance that your connection dies; in this case your ship will automatically uncloak before warping off and being logged off. Finally, I suspect that it is possible for people warping to the wormhole to uncloak you if they happen to be close to you as they come out of warp.
So, here's how to do it. If you are in the system where you want it, but not on grid with the wormhole, warp to it at 70km or 100km. A ship with a covert ops cloak should be cloaked while warping. A ship with a lesser cloak must wait until it lands on grid. In either case, immediately orbit the wormhole at your current distance so that you are no longer directly in line with a known celestial. Finally, use Look At to get a short perceptual distance from the wormhole. After a while of orbiting, you are likely to be out of line with everything and you can stop.
If you are on grid with the wormhole usually it is because you just came through it. In this case, use "Orbit at 30" to get distance, and before cloaking, pulse your microwarp if you have one to speed it up. 30km here is an example of convenience. If you have spare time, you can move further out, but this means stopping by hand. Alternatively, you can bounce off a nearby celestial and come back at range.
Examples of Use
There can be many reasons for wanting to know when a ship transits a wormhole. Perhaps the most common is monitor your home system while you do PI with an alt. PI runs only take a short while, and are relatively low risk anyway. Have two accounts; one can picket your static wormhole while the other does PI, then you swap places.
You can use pickets while doing PVE, i.e. killing sleepers for ISK. Generally, it is better to zip up than picket, but you won't always want to do that.
Another common situation where you will picket wormholes (often more than one) is when your corp is transporting stuff to and/or from known space. In this case you cannot be zipped up. Once you have a route and have determined that the systems along the route seem to be idle enough to move transports through, you want to keep watch on them to make sure they have stayed that way. So, you picket the wormholes in the direct path out. If you have additional pilots, you might also picket other wormholes in the chain.
A less common use is when you have found a target and are going to gank him. Here, you don't want to be interfered with, or at least you need to know when your target's friends are coming to help him. Perhaps your target is in system B with a connection to system C, and you've determined that C is his home. In that case you might picket the B->C wormhole before attempting your gank.