Baker was born in Groesbeck, Texas, the son of Edna (née McDonald) and Doyle Charles Baker. In 1964 he appeared on stage in "Marathon '33" at the ANTA Theatre in New York City. He got his start in acting as an uncredited character in the 1967 film Cool Hand Luke, and as an illiterate vending machine robber in a 1969 episode of the TV series Mod Squad, but his real beginnings came when he scored the role of Steve McQueen's younger brother in the film Junior Bonner, directed by Sam Peckinpah. He later starred in the 1973 film Walking Tall, directed by Phil Karlson, also starring in the film maker's final work Framed, two years later. Baker was offered a cameo in the remake (which had nothing to do with Sheriff Buford Pusser) but declined the offer.

His work in 1973's Charley Varrick may remain Baker's most memorable success. Baker was praised for a courageous and offbeat portrait of the sadistic hitman Molly. The film starred Walter Matthau as the bank robber Varrick, and won a British Academy Award.

Baker has given many outstanding performances in a career spanning four decades. In 1980, he became the first actor[citation needed] to receive $1,000,000 to star in a television series—the short lived police drama Eischeid where he played the character of tough and brilliant New York City Police Department Chief of Detectives Earl Eischeid.

He was "The Whammer," a mighty baseball slugger clearly modeled after Babe Ruth, in the 1984 baseball drama The Natural that starred Robert Redford. In a scene, the Whammer takes three swings at pitches from the young Roy Hobbs to try to impress a mysterious woman they have met on a train.

In 1985, he portrayed the corrupt Chief Jerry Karlin in Fletch. In the UK, Baker is probably best known as CIA agent Darius Jedburgh from the BBC Television drama serial Edge of Darkness. He was nominated for "Best Actor" by the British Academy Television Awards, losing to his co-star Bob Peck.

Martin Scorsese directed him as a private detective in 1991's Cape Fear, hired by a man (Nick Nolte) whose family is being threatened by a psychopathic ex-convict (Robert DeNiro).

While actor Carroll O'Connor was undergoing heart bypass surgery, Baker took his place on the television series In the Heat of the Night. Baker appeared as Captain Tom Dugan, a retired police captain who filled in while O'Connor's character was away at a police convention.

Baker's films Mitchell and Final Justice were lampooned on the movie-mocking television series Mystery Science Theater 3000. According to the Mystery Science Theater 3000 Amazing Colossal Episode Guide, Baker was angered by the treatment of Mitchell and threatened to physically assault any of the MST3k cast or crew if he ever met them. Kevin Murphy, MST3k's Tom Servo and one of the show's head writers, has suggested that Baker was probably joking, and it did not prevent them insulting Baker and his work further in the later commentary to Final Justice.

In 1987, Baker got the role of the villain Brad Whitaker in the Bond film The Living Daylights, starring Timothy Dalton as James Bond. In 1995 and 1997 Baker returned to the series, this time playing a different character, CIA agent Jack Wade, in GoldenEye and Tomorrow Never Dies with Pierce Brosnan as Bond. This makes Baker one of only two actors to appear as both a Bond ally and a villain, the other being Charles Gray who appeared as Henderson in You Only Live Twice and as Blofeld in Diamonds Are Forever.

The character of Wade is similar to that of CIA agent Darius Jedburgh, played by Baker in the 1985 serial Edge of Darkness. This serial was directed by Martin Campbell, who also cast Baker as Wade in GoldenEye.