Can I use target display mode on iMac(Retina 5k, 27-inch, 2017) running macOS Monterey?

We get your question multiple times a day. The answer is NO, Target Display Mode died 10 years ago in 2014. Your iMac is a computer it is NOT an external display for other devices. If your MBA needs an external display please buy one! Why, simple:

• No expensive cabling needed
• Dead simple to connect
• Dead reliable
• Low cost and less than $100 for an external display, of course you can also spend $1500 if you want a top of the line 5K external display. However for about $300 you can have a VERY nice external display

No. Target Display Mode went away when the first 27" 5K Retina iMac came out in Late 2014, and never returned. Apple also added restrictions on it which mean that Macs released in 2020 or later (such as your M3 MacBook Air) do not support using ANY iMac as a Target Display.

Reference: Use your iMac as a display with target display mode - Apple Support

-----

There is a lower-quality method of using one Mac as a second display for another, called AirPlay to Mac – but your 2017 iMac is too old to act as an AirPlay Receiver. (You might be able to AirPlay in the other direction – using your M3 MacBook Air as an AirPlay screen for the 2017 iMac.)

Reference: Continuity features and requirements for Apple devices - Apple Support

-----

There are third-party workarounds such as Luna Display (Astropad) and Duet Display. These are not as good as a hardware-supported connection to a 27" 5K Apple Studio Display. From an article on the Astropad site: "Like any product, Luna Display does have limitations and it’s not recommended for high-motion content. For example, I wouldn’t recommend using Luna Display for gaming or video editing."

There are third-party apps that can do this in a fashion, such as the oft-mentioned Luna Display, but this is suitable only for low-motion applications, like for instance tool palettes for an app, or other things where lag and occasional drops aren't an issue. Even then you need Ethernet or a really fast, reliable Wi-Fi. As rkaufmann suggested, I think you'll be much happier just getting an external display.

Regards.