Take with you: 

Spare batteries or a spare GPS device (such as a smart phone).

Spare flashlight, not just spare batteries. Changing batteries in the dark can be tricky.

Hiking stick: use it to roust out the wildlife before reaching into holes and under ledges, and hold it up in front of you while bushwhacking to catch the spider webs.

Insect repellent

Water
 

Do:

Know the closing times of any parks you'll be caching in.
Carry your phone.

If no other adult is caching with you, tell someone where you are going and when they should expect you to check in with them. Make sure they're prepared to call for help if you don't check in.

Make sure your GPSr's track log is turned on, to help you get back out.

Mark waypoints at your car, at the trailhead, and other points along the way.

Sight down the length of your flashlight (or use a headlamp) when you're looking for reflectors: there may be a drastic difference in brightness of the reflection if you hold your flashlight at waist or even chest level. See warning below about headlamps.

Try not to call attention to yourself: keep lights and noise down when you're caching in a cemetery or near anyone's or any business' private stuff.
 

If you have any suggestions for adding to this list, please email me. (You'll need to be logged into geocaching.com to use this link.)

Comments:

A headlamp is great for keeping your hands free if you're alone while logging the cache, but it will attract clouds of insects (or a few honkin' big ones) right to your face.

If you're out at night with the Tush Brothers, be sure to turn your headlamp off before hopping back into the back seat, to avoid comments.

Don't count on using your iPhone as your GPSr and flashlight. Even if you can get both apps to run at once, the battery won't last long.
Encounters with wildlife can be much more exciting in the dark. In urban areas and some wooded areas, surprise encounters with the homeless can be equally stimulating, as I know from experience.