Real-time drawing synchronization between PKCanvasView on iPad and external screen

Question

Situation: I'm developing an iPad app using SwiftUI, and I have an external screen connected to the iPad. In my app, I want to provide a drawing feature using PKCanvasView. The goal is to allow users to draw on the iPad screen and have the drawing display in real-time on the external screen. How can I achieve this synchronization between the PKCanvasView on the iPad and the external screen?

I have tried using an ObservableObject and then accessing the canvas property to pass it to the ExternalCanvas View

When I try this, I can draw on the iPad but when I connect an external monitor it removes the PKCanvasView off the iPad screen and puts it on the external monitor.

EDIT: I have also been back and forward with ChatGPT for almost 3 hours trying to get a resolution, but it was unable to suggest anything, mainly because it didn't seem to grasp the concept of what I was trying to achieve and they more it tried the further it got from the desired outcome

class SharedData : ObservableObject {
   @Published var canvas : PKCanvasView = PKCanvasView()
}

import SwiftUI
import PencilKit
struct ExternalView: View {
    @EnvironmentObject var shared : SharedData
    var body: some View {
            ExternalCanvas(canvasView: shared.canvas)
    }
}

struct ExternalCanvas: UIViewRepresentable {
    @State var canvasView: PKCanvasView

    func makeUIView(context: Context) -> PKCanvasView {
        canvasView.drawingPolicy = .anyInput
        canvasView.tool = PKInkingTool(.pen, color: .black, width: 15)
        return canvasView
    }

    func updateUIView(_ canvasView: PKCanvasView, context: Context) { }
}

Answer 1

PKCanvasView is a subclass of UIView. A UIView can only have a single superview. So you cannot use the same instance of PKCanvasView in two places (built-in screen and external screen) at the same time. You need one PKCanvasView for each place where you want to show the drawing, and you need to keep the drawing property of all the PKCanvasViews in sync.

PKCanvasView only updates its drawing when a stroke ends, so you cannot quite get all the views to synchronize in "real-time". This is what you get:

First, I wrote a wrapper for PKCanvasView:

struct PKCanvasViewWrapper: View {
    @Binding var drawing: PKDrawing

    var body: some View {
        Rep(spec: self)
    }
}

I like to keep my UIViewControllerRepresentable types private, so that wrapper is just a View, and it uses a private type named Rep:

extension PKCanvasViewWrapper {
    typealias Spec = Self

    fileprivate struct Rep: UIViewControllerRepresentable {
        var spec: Spec

        func makeUIViewController(context: Context) -> Controller {
            return Controller(spec: spec)
        }

        func updateUIViewController(_ controller: Controller, context: Context) {
            controller.update(to: spec)
        }
    }
}

The Controller type is a UIViewController that manages the PKCanvasView:

extension PKCanvasViewWrapper {
    fileprivate class Controller: UIViewController {
        private var spec: Spec
        private let canvas: PKCanvasView

        init(spec: Spec) {
            self.spec = spec

            canvas = PKCanvasView()
            canvas.drawingPolicy = .anyInput
            canvas.tool = PKInkingTool(.pen)

            super.init(nibName: nil, bundle: nil)

            self.view = canvas
            canvas.delegate = self
        }
        
        required init?(coder: NSCoder) { fatalError() }

        func update(to spec: Spec) {
            self.spec = spec

            if canvas.drawing != spec.drawing {
                canvas.drawing = spec.drawing
            }
        }
    }
}

The Controller type conforms to PKCanvasViewDelegate so that the canvas can notify it when the drawing changes:

extension PKCanvasViewWrapper.Controller: PKCanvasViewDelegate {
    func canvasViewDrawingDidChange(_ canvasView: PKCanvasView) {
        if canvasView.drawing != spec.drawing {
            spec.drawing = canvasView.drawing
        }
    }
}

In a real app, you might want to add more properties to the PKCanvasViewWrapper type to specify how to configure the PKCanvasView. Then, in the Controller's update(to:) method, you'd need to use those properties to configure the PKCanvasView.

Anyway, I then wrote an ObservableObject type to hold the shared PKDrawing:

class AppModel: ObservableObject {
    @Published var drawing: PKDrawing

    init() {
        drawing = PKDrawing()
    }
}

I added an appModel property to my AppDelegate to make it available to all scenes:

@main
class AppDelegate: UIResponder, UIApplicationDelegate {
    let model = AppModel()

    func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?) -> Bool {
        return true
    }

    func application(_ application: UIApplication, configurationForConnecting connectingSceneSession: UISceneSession, options: UIScene.ConnectionOptions) -> UISceneConfiguration {
        return UISceneConfiguration(name: "Default Configuration", sessionRole: connectingSceneSession.role)
    }
}

Then I wrote a SwiftUI view to appear on the internal screen:

struct InternalRootView: View {
    @ObservedObject var model: AppModel

    var body: some View {
        VStack {
            Text("Internal Screen")
            PKCanvasViewWrapper(drawing: $model.drawing)
                .frame(width: 300, height: 300)
                .border(Color.gray)
        }
    }
}

and I wrote a scene delegate to create that view and show it in a window:

class InternalSceneDelegate: UIResponder, UIWindowSceneDelegate {
    var window: UIWindow?

    func scene(_ scene: UIScene, willConnectTo session: UISceneSession, options connectionOptions: UIScene.ConnectionOptions) {
        guard
            let appDelegate = UIApplication.shared.delegate as? AppDelegate,
            let scene = scene as? UIWindowScene
        else { return }

        let window = UIWindow(windowScene: scene)
        window.rootViewController = UIHostingController(rootView: InternalRootView(model: appDelegate.model))
        window.isHidden = false
        self.window = window
    }
}

I copied the view and scene delegate for the external screen, just changing "Internal" to "External" everywhere.

Finally, I updated the "Application Scene Manifest" in the Info tab of my app target:

Application Scene Manifest
  Enable Multiple Windows NO
  Scene Configuration (2 items)
    Application Session Role (1 item)
      Item 0 (Default Configuration)
        Delegate Class Name: $(PRODUCT_MODULE_NAME).InternalSceneDelegate
        Configuration Name: Default Configuration
    External Display Session Role Non-Interactive (1 item)
      Item 0 (Default Configuration)
        Delegate Class Name: $(PRODUCT_MODULE_NAME).ExternalSceneDelegate
        Configuration Name: Default Configuration

You can find the full project in this github repo.