Wow. What a dilemma. I don't have the best answer. But will list out some pros and cons for each.
The first are con's against a car. Victoria is hell to navigate. Sure, an easy 30 minute drive from the ferry to downtown where your hotel is probably located. But in general, the roads wind and change names, etc. I think it is partially because back in the day, they had a British round-a-bout system that is now converted to traffic lights. With the latter you would have parallel streets running into 2 street intersections. In the former why not 4 or 6 roads heading to a round-a-bout. Plus it is an island with peninsulas, bridges, and coastlines; the roads can't be straight. So, when you add an an elderly population driving, plus other tourists driving, it may not be a vacation to drive. Other than Butchard Gardens, there are a lot of attractions in the downtown area, so a car is not need. And basically, all transit routes go downtown, so if your hotel isn't downtown you will probably be able to get there easy. Finally, the bus drives right onto the ferry, you don't have to deal with the line up. AND there will be a line up at the ferry.
PCL is the bus that goes from the airport, right onto the ferry and to downtown Victoria. It's a long trip, and the fewer times you transfer your luggage the better. Their link is attached. Problems? The ride takes forever, it leaves every 2 hours and the last one is 7:40 PM, probably getting you to downtown Victoria 11:30ish. I guess realistically, you can only leave by car about half an hour later than that last bus to make it. You may want to see if the PCL return trip via Buchard Gardens works for you. Looks to be a smooth option, but the timing has to be right for your other travel plans. Renting a car and a well timed ferry reservation could shorten that trip by 90 minutes. It's not guaranteed, there are lots of variables, especially if you skip the ferry reservation. But my bet is a car is smother and faster. And the cost is about the same for caching the ferry with a car as it is for 2 taking the bus. You still have costs of the car, but if 3 or 4 are traveling, the car becomes cheaper. Even with 2, if you happen on a good car rate like a weekend or week deal, the car option can be cheaper.
The double decker bus is the other tour option for hitting Buchard. It is also attached. It is $50 each. Buchard costs $23 each. So again, you can ask for specific driving directions from the hotel, rent a car for the day, and get there quicker, cheaper and freer. The thing I hate about a bus is the freedom. With your own car you can leave earlier if you get board, or stay until it closes if you love the place.