Three things I've been doing lately are running an OSE campaign in Neverland, a 3rd edition campaign in a brutal wilderness, and an in-depth study of the AD&D rules. With the difference in rulebook size, I've been thinking about... well, rulebook size.
Good book sizes
I quite like the compact size of the OSE rulebooks. They feel very "tight" and don't take up much table room. You might wonder if the small page size means that you'll end up flipping back and forth between pages for multi-page tables or something, but B/X is so devoid of rules and lengthy tables that it doesn't happen. The book's decently thick for its content, which can be an issue in its own right, but it's a worthy sacrifice to make the book easy to scan and reference during play. The perfect book size was selected for OSE.
The OSE book size would have been impossible for AD&D. The books would be four feet tall and impossible to reference. Imagine the tables! The AD&D books are much larger than OSE in width and height, but are far thinner – even the DMG. Most importantly, tons of rules fit on its two-column, two-page spread. The books are light enough to hold in one hand. The perfect book size was selected for AD&D.
The 3rd edition books are more or less the same size as the AD&D DMG. At least, width-and-height-wise. If you take a look at the actual page content, enormous space is wasted on the 3rd edition pages, which are themselves slightly smaller to boot. A clear win for AD&D. Though I'd still consider the 3rd edition books to be more "good" than "bad," dimensionally.
Bad book sizes
One of my big pet peeves is when referee material is placed in the players' book. Pathfinder is one offender that comes to mind. A massive $50 "Core Rules" tome that contains some GM info, plus a thin $50 "Gamemastery Guide" is just a shit way to do things. Make reference books reference-able. If your book is tall, wide, and 500 pages deep it's going to be a pain to use at the table. If half the GM info is in one book and half is in another book, they're going to be a pain to use at the table.
"But then you can buy the entire game in one book!"
I could put all my furniture in one room too but I don't do that because it would be stupid and awful. Of course, this quip ignores the fact that one book is not the entire game, you still need the Bestiary, and part of the game is in the Gamemastery Guide. But this isn't a Pathfinder dig post. That would be too easy.
For smaller RPGs, I hope people follow OSE's example. Those books feel great. No need for letter-sized-or-larger pages when you're working with a ruleset like that. Take the opportunity to make the books nice and compact!
For larger RPGs, I wish more people followed AD&D's example. Each page is used to great effect. Plus, the books are pretty lightweight, especially for the amount of content within. I suppose the reason we don't see a lot of books like these is because of the art and flourishes used in contemporary RPGs, such as the page decoration in the 3rd edition books.
If I end up printing my AD&D annotations, my feeling is a smaller paper size would be ideal. Perhaps something OSE-sized or a little larger. It shouldn't need large tables and would work best as a quick-reference handbook. As of right now, I'm writing annotations in order of the page number the related rule appears on in the AD&D book. Perhaps it would be better to sort by rule name or topic, though, with bold page headers to make it easy to flip through the book and find what you're looking for. Annotations sorted by source page number could be listed in the back. I guess that would make it more like a field guide than merely a list of compiled annotations. I wonder what would be best?