The Hindu Succession Act, 1956 (HSA) sets out how the property of a Hindu male dying intestate (i.e. without a will) would devolve
How will the property of a man who has died intestate and is survived by an elder brother having sons and a daughter, besides the sons of a predeceased elder sister, be divided? Do the sons of the late sister have equal rights as those of the elder brother?
—Name withheld on request
Assuming the family is Hindu by faith, the Hindu Succession Act, 1956 (HSA) sets out how the property of a Hindu male dying intestate (i.e. without a will) would devolve.
Your query infers that his parents are not alive/predeceased him, and also that he is not survived by his wife and children (i.e. his Class I heirs under the HSA).
As there is no Class I heirs alive, the property of the deceased would be distributed among the Class II legal heirs. Hence, the surviving siblings of such deceased brother (as category 2 - Class II legal heirs) would take an equal share in the deceased brother’s estate, to the exclusion of other relatives.
No share in the late brother’s estate would devolve upon the surviving brother’s heirs (i.e. his daughter and sons), as long as such brother is alive.
Likewise, since the deceased is survived by his brother, the late brother’s estate will not devolve upon the heirs of the predeceased elder sister (i.e. her sons).