They found alcohol use was associated with reduced size of the right hippocampus - a part of the brain linked to memory and navigation.
The research, which was conducted in the United Kingdom, defined moderate drinking as equal to 8 to 12 small glasses of wine, bottles of beer, or shots of liquor each week in the United States.
"We were surprised that the light to moderate drinkers didn't seem to have that protective effect", said study co-author Dr. Anya Topiwala, a clinical lecturer in the Department of Psychiatry at the University of Oxford.
However, the long term effects of moderate drinking on cognitive function and brain activity are less well understood.
Boozing, even at moderate levels, harms the brain, scientists found.
"People should be skeptical of the idea that it's actually healthy, and treat alcohol with respect", said Tim Stockwell, director of the Center for Addictions Research at the University of Victoria, in British Columbia, Canada. Also, majority were subjected to MRI scans.More news: Oregon tops LSU 4-1, will play Oklahoma in WCWS semifinals
Consistently drinking between 14 and 21 units a week - the equivalent of six to nine pints of beer or the same number of 175ml glasses of wine - results in brain shrinkage and cognitive decline in old age, scientists found.
Over the 30-year period, the researchers measured the participants' alcohol intake and performance in cognitive tasks.
"The types of scans we did allowed us to look closely at the brain".
In the latest study published in the BMJ, researchers from the University of Oxford and University College London looked at the impact of moderate alcohol consumption on the brain. On the other hand, moderate alcohol consumption has been associated with a reduction in brain volume and gray matter atrophy.
In January 2016, the Government changed its guidance on drinking and urged both men and women to drink no more than 14 units each week.
They also found that very light drinking - classed as drinking between one and six units a week - had no protective effect compared to abstinence.More news: Trump waives law requiring US move its embassy to Jerusalem
It's interesting to note however that despite the structural changes to the brain observed on the scans, the participants did not yet show significant signs of cognitive decline.
"We think, though, this study asks some really important questions in an area that does have a knowledge gap", Topiwala said. The test gives people a minute to name as many words as they can, beginning with a particular letter.
"Whilst we attempted to control for this, it is possible that higher IQ might be associated with other behaviours we have not accounted for, such as dietary factors". Finally, "unsafe" drinkers were classified as those that drunk in such a way that they wouldn't fit the classifications mentioned above.
There is no health reason for anyone who does not drink to take up drinking, said Brian Downer, an assistant professor in the division of rehabilitation sciences at the University of Texas Medical Branch at Galveston. If you are the one to closely monitor your alcohol consumption and consciously limit your alcohol intake, researchers opine that it is not good enough to guard you from the ill-effects of alcohol.
'This is just one study, and the researchers admit they can not say for certain that alcohol is causing the effects that they have observed.
The U.S. Centers for Disease Control and Prevention has more on alcohol and health. "Hopefully this research will contribute to a greater understanding of true safe limits for alcohol consumption that ensure protection from future dementia".More news: MTV will livestream Sunday's Manchester benefit concert