In order to further improve the accuracy of image segmentation for visual obstacle avoidance of watercraft, an image segmentation algorithm for watercraft obstacle based on fusion of modified OTSU and median filter (VSAIMOAMF) is proposed. The original image is transformed from RGB model to Y, Cb, Cr chroma value modified model by using correction coefficient. The threshold segmentation of modified OTSU is carried out. The segmented binary image is processed by adaptive median filtering and noise reduction. Three water obstacle recognition algorithms are tested. The results show that, compared with the weighted Otsu algorithm and the improved threshold segmentation algorithm, the algorithm can stabilize the proportion of the detection target area to more than 80%, and reduce the proportion of the interference noise area to 28.5%, which shows that the algorithm is effective and feasible.