When the end of the 20th century came, some aspects of jazz began to be given the status of a classical form. In the reassessments that followed, the work of the American pianist and composer Dave Brubeck, who has died aged 91, was a major beneficiary. He was a figure simultaneously feted and mugged by ecstatic fans and infuriated purists during the years between 1954 and 1966 – the time when his catchiest and most deftly composed records were pop hits.

Like the Modern Jazz Quartet, which enjoyed similar commercial success in that period, Brubeck's music flattered and engaged the young white middle-class, and particularly the student population, much as the classical-sounding clarinettist Benny Goodman's work had done in the mid-1930s. Brubeck intertwined jazz swing with time-signatures that looked like algebra, and mingled standard song-forms with rondos and fugues. All kinds of music fans who would have hated to be seen with a jazz album owned Brubeck records in the 60s, just as they own Diana Krall, Jan Garbarek or Keith Jarrett discs today.

But if Brubeck's success, and the repertoire that achieved it, could be fighting talk among music-lovers 40 years ago, now time, and the eclecticism and fluid collaborations of a shrinking world, have healed his estrangement from the jazz audience. Brubeck's pieces are now recognised for the harmonically subtle, melodically devious and original works they are, and his most classically oriented works (such as the soft-winds Bach tribute Chorale) as triumphs in a treacherous territory in which short-changing jazz or dumbing-down symphonic composition is very hard to avoid. The Brubeck debate eventually vanished into the archives, and his real gifts – as a composer, and a charter of new rhythmic waters as inventive as the brilliant bebop drummer-composer Max Roach – came to be appreciated for what they always were.

Unlike Goodman and his college audience triumphs of the 1930s, Brubeck discovered his jazz in the postwar world – in a very different climate, which initiated the unusual chemistry of his music by a very different route. Jazz, pop and dancing were synonymous in the 30s. But Brubeck emerged a decade later, after the more cerebral and exploratory modernist idiom of bebop had profoundly influenced the music.

To make jazz popular again, to haul it out of the bare-bulb, hipster-subculture cellar it had holed up in during the late 1940s, would require a different approach. Brubeck, who grew up on a California ranch and initially trained as a vet, certainly made no such opportunistic calculations in the beginning, and wanted only to pursue his abiding passion for music any way he could. However, the populist approach found him in the end, whether he was looking for it or not. Born in Concord, California, he was originally trained in classical music, at first by a piano-playing mother. His environment cut him out to be a cowboy more than a musician – and with two older brothers on their way to music college, he was initially happy to embrace the alternative of working on the land himself. Though he resisted leaving the family ranch, his parents persuaded him to enrol as veterinarian major at the College of the Pacific.

Brubeck's musical enthusiasms then overtook him, and he switched courses after a year – to the mingled delight and anxiety of the music faculty, which welcomed someone clearly cut out to be one of its most naturally gifted students, but whose notation-reading was so poor they made him promise never to teach music before they had let him graduate. During this period Brubeck led and played in jazz bands most nights of the week, and also met Iola Whitlock, who ran the weekly campus radio show. Brubeck proposed to her after a two-week relationship, and she survives him, along with four sons and a daughter.

During the second world war, Brubeck was given an army band to run on a tour of Europe in 1944. His superior officer, a jazz fan, repeatedly intervened to prevent the musician being sent to the front. After demobilisation, he studied at Mills College with the classical composer Darius Milhaud, who revealed the intricacies of polyrhythm and polytonality to him, and influenced his music for life, telling him that if he wanted to express America, he would always use the jazz idiom.

Brubeck then founded an experimental collective, the Jazz Workshop Ensemble. It was dedicated to exploring forms of jazz less hidebound by orthodox "swing" and Tin Pan Alley-derived harmonic structures. In 1951 he formed his first quartet, including a feathery-sounding alto saxophonist called Paul Desmond, a confirmed disciple of the undemonstrative, dynamically restrained white "cool school" variations on bebop whom he had briefly worked with in San Francisco in 1947. The pianist set up his own record label, Fantasy Records, and released Jazz at Oberlin (1953), the quartet's first album. This exploration of live recording, rare for the time, secured a deal with Columbia Records – and the ensuing Jazz Goes to College (1954) sold over 100,000 copies. The success made Brubeck the first jazz musician to be featured on the cover of Time magazine, which said he was ushering in the birth of a new kind of jazz age in the US.

In the late 1950s the first quartet lineup, with bassist Norman Bates and drummer Joe Dodge, was transformed by the remarkable drummer Joe Morello and the bassist Gene Wright, and the most popular and influential Brubeck quartet was born. The familiar four-on-four metre of straight-ahead jazz time was augmented by complex tempos like 9/8 (as in the engaging Mozartian Blue Rondo a la Turk) and 11/4, though the improvising sections of Brubeck's pieces frequently loosened into regular swing, which ingeniously balanced their appeal.

The album Time Out (1959) turned out to be the group's biggest landmark, unleashing the first million-selling post-bebop jazz records with singles of Blue Rondo and the Desmond composition – triggered by a 5/4 Morello drum exercise – Take Five. Between 1959 and 1965, the Quartet won Down Beat magazine's readers' poll five times and was Playboy readers' favourite jazz group for 12 years running, from 1957 to 1968. By the early 1960s, the New Yorker announced that the quartet was "the world's best-paid, most widely travelled, most highly publicised, and most popular small group now playing improvised syncopated music".

But this success had not come without reservations in the jazz world. Brubeck was on the wrong side of the purists almost as soon as his discs started to become hits – for what were seen by some as three betrayals. First, and maybe worst, he made money, which was a form of notoriety usually regarded as a sell-out by hardline hipsters. Second, his conspicuously complex tempos paraded cleverness and a fondness for European classical devices at a time when black American jazz was dumping much of its formal baggage, and fiery, impassioned and unpredictable improvisers such as Sonny Rollins and John Coltrane were on the rise. Third, he was portrayed by the cognoscenti as wasting the talents of a truly great improviser in Desmond, his lyrical and delicate alto saxophonist.

Such generally perspicacious writers as the British critic Benny Green were merciless with Brubeck. But the band was a huge success all around the world, and toured constantly. The jazz-loving American comedian Mort Sahl once remarked of American cold-war foreign diplomacy that "After John Foster Dulles visits a country, the State Department sends the Brubeck Quartet in to repair the damage."

The quartet finally disbanded in 1967, rejoining only once, in 1976, for a 25th anniversary tour. Brubeck branched out, concentrating increasingly on large-scale composition, writing ballets – Points on Jazz (1960) entered the repertory of the American Ballet Theatre – a mass, various cantatas, and combinations of jazz musicians and symphony bands. He also began performing with his highly musical sons: Darius, a keyboard player; Chris, a trombonist/bassist; Danny, a drummer; and latterly the youngest, cello-improviser Matthew. He also worked effectively with the saxophonist Jerry Bergonzi in the 80s, and the uncannily Desmond-like Bobby Militello in the 90s.

Brubeck's landmarks, awards and citations became too numerous to count – he played for presidents Kennedy, Johnson, Reagan and Clinton, appeared at the Reagan-Gorbachev Moscow summit in 1988, and composed a score for Pope John Paul II's visit to San Francisco in 1987.

All his life, Brubeck continued to regard himself as "a composer who plays the piano". Though much was made of his piano-playing by his early fans, Brubeck's solos relied heavily on riff-like block chords and rather relentless dynamics. They became more varied and unpredictable in the later stages of his career and remained so into his 80s. But Brubeck's real achievement was to blend European compositional ideas, very demanding rhythmic structures, jazz song-forms and improvisation in expressive and accessible ways. His son Chris was to tell the Guardian, "when I hear Chorale, it reminds me of the very best Aaron Copland, something like Appalachian Spring. There's a sort of American honesty to it."

However difficult Brubeck's pieces became, they could still be whistled at the bus-stop. That's likely to go on happening for a long time.

• David Warren Brubeck, composer and pianist, born 6 December 1920; died 5 December 2012