The major cause for the cumbersome transfer situation is the current BART alignment with 1) a stub-end station adjacent to International Terminal, 2) a wye outside the airport by the freeway, and 3) a Caltrain/BART transfer station at Millbrae.
Although the wye permits direct train service between Millbrae and SFO, it complicates rail operation and BART decided not to run trains between SFO and Millbrae (during weekday daytime) due to operational and cost issues.
PR2000 (before renaming to BayRail Alliance) fought for a BART extension alternative that includes a single transfer station between BART, Caltrain, and AirTrain. Under that scenario, all BART and Caltrain riders to the airport would have to transfer to AirTrain. Transfer from regional rail systems to an airport-run rail system is not unusual. It can be found at New York’s JFK and Newark airports.
Even if BART were to go inside the airport, a transfer station between BART and Caltrain can be built in San Bruno, where they currently run side by side (Caltrain on ground level, BART is right next to Caltrain in a subway).
Quentin Kopp, who was a state senator in the mid-1990s, was the chief advocate for the current set up. He made a sound-bite political issue (by making the transfer to AirTrain more daunting than it would have) by placing Proposition I on the ballot in 1994, which called for construction of a BART station inside the airport. It was approved by San Francisco voters. Kopp had no concern about making a connection with Caltrain efficient and believed (and still does) that BART should replace Caltrain.
Because of the decision on BART alignment, Caltrain advocates convinced the agency to perform a separate study on the direct connection to AirTrain, but the study was quietly discontinued in 2001.
Before the BART extension opened, there was a free shuttle between Caltrain at Millbrae to SFO. It ran every 15 minutes 7 days a week.
When the BART extension first opened in 2003, there was a separate line operating between Millbrae and SFO, along with a direct line from SF to SFO and another to Millbrae. The BART shuttle operated every 20 minutes (as supposed to every 15 minutes like all other lines) due to line configuration and union contract issues. The BART shuttle was not timed with Caltrain departures and arrivals and riders had to walk up and over from Caltrain to reach BART (unlike the trains to SF which served the platform adjacent to Caltrain). The shuttle discontinued a year later due to large operating deficits and low ridership. Over the next few years, direct service between Millbrae and SFO was provided by the main line to San Francisco. That was eliminated in 2007 (for weekday daytime only) when BART decided to run another main line from SF to Millbrae without going through SFO.
Because of high cost and lower than expected ridership, BART and SamTrans made a number of operational decisions that were quite “dramatic.” In 2009, when BART imposed a $4 surcharge on all riders entering and and exiting the SFO station, including airport employees, SFO protested the surcharge on employees by running a free shuttle bus between SFO and Millbrae for about a year until SFO and BART compromised on a discount program.
To this day, SamTrans does not plan to provide any shuttle or any bus route between Millbrae and SFO for not wanting to “compete” with BART. This is an prime example of transportation planning failure (especially driven by politicians with ill-intents). Aside from the politics associated with the planning of BART, the design resulted in operational difficulties that we “will have to live with indefinitely.” 
Academic reports on the extension:
- Page 14, Cast Study Report: San Francisco International Airport BART Extension, Mineta Transportation Institute