The automatic classification of electrocardiograms (ECGs) plays a crucial role in the early diagnosis of cardiovascular diseases. In recent research, deep neural network (DNN)-based methods have garnered significant attention due to their exceptional feature extraction capabilities. However, these methods face challenges in dealing with the complex inter- and intra-class dependencies inherent in different arrhythmias. We propose a classification model for electrocardiograms based on the Graph Convolutional Neural Network (EGCNet) to address this. In our EGCNet framework, we first employ the row and column 1-dimensional residual convolution to learn intra-class dependency. Then, we construct a hierarchical graph structure to model the inter-class dependency. In such a graph, each node represents one type of disease, and the edges describe the relationship between diseases. To reduce the complexity of inter-class dependency, we build a hierarchical graph structure by treating similar nodes as the same type. Thus, we can learn the discriminant dependency in the higher-level graph space and feed this high-level knowledge back to the lower-level graph space. Finally, we evaluated our method using publicly available datasets from the PhysioNet/Computing Challenge 2020 ECG data. For instance, our model achieved robust performance in the 24-label experiment with a precision of 0.753 and an F1 score of 0.722.